PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types
of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in
PKCS#11 mode using OpenSC to handle system login. libpam-pkcs11 is a pretty
easy to use module that will let you log into your system locally using a
PKCS#11 token locally.
One of the least documented things, though, was how to use an OpenSC PKCS#11
token in Chrome. First, close all web browsers you have open.
About a year ago, I bought a Projector after I finally admitted that I could
actually use a TV in my apartment. I settled on buying a
ViewSonic PJD5132.
It was a really great value, and has been nothing short of a delight to own.
I was always a bit curious about the DB9 connector on the back of the unit,
so I dug into the user manual, and found some hex code strings in there. So,
last year, between my last gig at the
Sunlight Foundtion and
USDS, I spent some time wandering around the US,
hitting up DebConf, and exploring Washington
DC. Between trips, I set out to figure out exactly what was going on with my
Projector, and see if I could make it do anything fun.
So, I started off with basics, and tried to work out how these command codes
were structured. I had a few working codes, but to write clean code, I'd be
better off understanding why the codes looked like they do. Let's look at
the "Power On" code.
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D
Some were 10 bytes, other were 11, and most started with similar looking
things. The first byte was usually a 0x06 or 0x07, followed by two
bytes 0x14 0x00, and either a 0x04 or 0x05. Since the first few bytes
were similarly structured, I assumed the first octet (either 0x06 or 0x07)
was actually a length, since the first 4 octets seemed always present.
So, my best guess is that we have a Length byte at index 0, followed by
two bytes for the Protocol, a flag for if you're Reading or Writing (best
guess on that one), and opaque data following that. Sometimes it's a const
of sorts, and sometimes an octet (either little or big endian, confusingly).
Right. OK. So, let's get to work. In the spirit of code is data, data is code,
I hacked up some of the projector codes into a s-expression we can use later.
The structure of this is boring, but it'll let us both store the command
code to issue, as well as define the handler to read the data back.
(setv*commands*; function type family control'((power-onnilnil(0x060x140x000x040x000x340x110x000x000x5D))(power-offnilnil(0x060x140x000x040x000x340x110x010x000x5E))(power-statusconstpower(0x070x140x000x050x000x340x000x000x110x000x5E))(resetnilnil(0x060x140x000x040x000x340x110x020x000x5F))...
As well as defining some of the const responses that come back from the
Projector itself. These are pretty boring, but it's helpful to put a
name to the int that falls out.
After defining a few simple functions to write the byte arrays to the serial
port as well as reading and understanding responses from the projector, I could
start elaborating on some higher order functions that can talk projector. So
the first thing I wrote was to make a function that converts the command
entry into a native Hy function.
Cool. So, now we can import things like power-on from *commands* which
takes a single argument (serial) for the serial port, and it'll send a
command, and return the response. The best part about all this is you only
have to define the data once in a list, and the rest comes for free.
Finally, I do want to be able to turn my projector on and off over the network
so I went ahead and make a Flask "API" on top of this. First, let's define
a macro to define Flask routes:
I ll be at HOPE 11 this year - if anyone else will be around, feel free to send me an email! I won t have a phone on me (so texting only works if you use Signal!)
Looking forward for a chance to see everyone soon!
Around a year ago, I started hacking together a machine readable version
of the OSI approved licenses list, and casually picking parts up until it
was ready to launch. A few weeks ago, we officially announced
the osi license api, which is now
live at api.opensource.org.
I also took a whack at writing a few API bindings, in
Python,
Ruby,
and using the models from the API implementation itself in
Go. In the following
few weeks, Clint wrote one in Haskell,
Eriol wrote one in Rust,
and Oliver wrote one in R.
The data is sourced from a repo on GitHub,
the licenses repo under OpenSourceOrg. Pull Requests against that repo are
wildly encouraged! Additional data ideas, cleanup or more hand collected data
would be wonderful!
In the meantime, use-cases for using this API range from language package
managers pulling OSI approval of a licence programatically to using a license
identifier as defined in one dataset (SPDX, for exampele), and using that
to find the identifer as it exists in another system (DEP5, Wikipedia,
TL;DR Legal).
Patches are hugly welcome, as are bug reports or ideas! I'd also love more
API wrappers for other languages!
A while back, I found myself in need of two webservers that would terminate TLS (with different rules). I wanted to run some custom code I d written (which uses TLS peer authentication), and also nginx on port 443.
The best way I figured out how to do this was to write a tool to sit on port 443, and parse TLS Client Hello packets, and dispatch to the correct backend depending on the SNI name.
SNI, or Server Name Indication allows the client to announce (yes over cleartext!) what server it s looking for, similar to the HTTP Host header. Sometimes, like in the case above, the Host header won t work, since you ve already done a TLS handshake by the time you figure out who they re looking for.
I also spun the Client Hello parser out into its own importable package, just in case someone else finds themselves in this same boat.
The code s up on github.com/paultag/sniff!
Last week, I posted about python-sense,
and API wrapper for the internal Sense API. I wrote this so that I could
pull data about myself into my own databases, allowing me to use that
information for myself.
One way I'm doing this is by pulling my room data into an
InfluxDB database, letting me run time series
queries against my environmental data.
I'm able to run this on a cron, automatically loading data from the Sense
API into my Influx database. I can then use that with something like
Grafana, to check out what my room looks like over
time.
A while back, I saw a Kickstarter
for one of the most well designed and pretty sleep trackers on the market. I
fell in love with it, and it has stuck with me since.
A few months ago, I finally got my hands on one and started to track my data.
Naturally, I now want to store this new data with the rest of the data I have
on myself in my own databases.
I went in search of an API, but I found that the Sense API hasn't been published
yet, and is being worked on by the team. Here's hoping it'll land soon!
After some subdomain guessing, I hit on api.hello.is.
So, naturally, I went to take a quick look at their Android app and network
traffic, lo and behold, there was a pretty nicely designed API.
This API is clearly an internal API, and as such, it's something that
should not be considered stable. However, I'm OK with a fragile API,
so I've published a quick and dirty API wrapper for the Sense API
to my GitHub..
I've published it because I've found it useful, but I can't promise the world,
(since I'm not a member of the Sense team at Hello!), so here are a few ground
rules of this wrapper:
I make no claims to the stability or completeness.
I have no documentation or assurances.
I will not provide the client secret and ID. You'll have to find them on
your own.
This may stop working without any notice, and there may even be really nasty
bugs that result in your alarm going off at 4 AM.
Send PRs! This is a side-project for me.
This module is currently Python 3 only. If someone really needs Python 2
support, I'm open to minimally invasive patches to the codebase using
six to support Python 2.7.
Working with the API:
First, let's go ahead and log in using python -m sense.
$ python -m sense
Sense OAuth Client ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense OAuth Client Secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense email: paultag@gmail.com
Sense password:
Attempting to log into Sense's APISuccess!Attempting to query the Sense APIThe humidity is **just right**.The air quality is **just right**.The light level is **just right**.It's **pretty hot** in here.
The noise level is **just right**.
Success!
Now, let's see if we can pull up information on my Sense:
>>>fromsenseimportSense>>>sense=Sense()>>>sense.devices()'senses':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'11a1','last_updated':1466991060000,'state':'NORMAL','wifi_info':'rssi':0,'ssid':'Pretty Fly for a WiFi (2.4 GhZ)','condition':'GOOD','last_updated':1462927722000 ,'color':'BLACK' ],'pills':[ 'id':'xxxxxxxxxxxxxxxx','firmware_version':'2','last_updated':1466990339000,'battery_level':87,'color':'BLUE','state':'NORMAL' ]
Neat! Pretty cool. Look, you can even see my WiFi AP! Let's try some more
and pull some trends out.
I plan to keep maintaining it as long as it's needed, so I welcome
co-maintainers, and I'd love to see what people build with it! So far, I'm
using it to dump my room data into InfluxDB, pulling information on my room
into Grafana. Hopefully more to come!
Happy hacking!
As some of the world knows full well by now, I've been noodling with Go
for a few years, working through its pros, its cons, and thinking a lot
about how humans use code to express thoughts and ideas. Go's got a lot of
neat use cases, suited to particular problems, and used in the right place,
you can see some clear massive wins.
I've started writing Debian tooling in Go, because it's a pretty natural fit.
Go's fairly tight, and overhead shouldn't be taken up by your operating system.
After a while, I wound up hitting the usual blockers, and started to build up
abstractions. They became pretty darn useful, so, this blog post is announcing
(a still incomplete, year old and perhaps API changing) Debian package for Go.
The Go importable name is pault.ag/go/debian. This contains a lot of utilities
for dealing with Debian packages, and will become an edited down "toolbelt"
for working with or on Debian packages.
Module Overview
Currently, the package contains 4 major sub packages. They're a changelog
parser, a control file parser, deb file format parser, dependency parser
and a version parser. Together, these are a set of powerful building blocks
which can be used together to create higher order systems with reliable
understandings of the world.
changelog
The first (and perhaps most incomplete and least tested) is a changelog file
parser.. This provides the
programmer with the ability to pull out the suite being targeted in the
changelog, when each upload was, and the version for each. For example, let's
look at how we can pull when all the uploads of Docker to sid took place:
funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")iferr!=nilpanic(err)allEntries,err:=changelog.Parse(resp.Body)iferr!=nilpanic(err)for_,entry:=rangeallEntriesfmt.Printf("Version %s was uploaded on %s\n",entry.Version,entry.When)
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
control
Next is one of the most complex, and one of the oldest parts of go-debian,
which is the control file parser
(otherwise sometimes known as deb822). This module was inspired by the way
that the json module works in Go, allowing for files to be defined in code
with a struct. This tends to be a bit more declarative, but also winds up
putting logic into struct tags, which can be a nasty anti-pattern if used too
much.
The first primitive in this module is the concept of a Paragraph, a struct
containing two values, the order of keys seen, and a map of string to string.
All higher order functions dealing with control files will go through this
type, which is a helpful interchange format to be aware of. All parsing of
meaning from the Control file happens when the Paragraph is unpacked into
a struct using reflection.
The idea behind this strategy that you define your struct, and let the Control
parser handle unpacking the data from the IO into your container, letting you
maintain type safety, since you never have to read and cast, the conversion
will handle this, and return an Unmarshaling error in the event of failure.
Additionally, Structs that define an anonymous member of control.Paragraph
will have the raw Paragraph struct of the underlying file, allowing the
programmer to handle dynamic tags (such as X-Foo), or at least, letting
them survive the round-trip through go.
The default decoder
contains an argument, the ability to verify the input control file using an
OpenPGP keyring, which is exposed to the programmer through the
(*Decoder).Signer() function. If the passed argument is nil, it will not
check the input file signature (at all!), and if it has been passed, any
signed data must be found or an error will fall out of the NewDecoder call.
On the way out, the opposite happens, where the struct is introspected,
turned into a control.Paragraph, and then written out to the io.Writer.
Here's a quick (and VERY dirty) example showing the basics of reading and
writing Debian Control files with go-debian.
packagemainimport("fmt""io""net/http""strings""pault.ag/go/debian/control")typeAllowedPackagestructPackagestringFingerprintstringfunc(a*AllowedPackage)UnmarshalControl(instring)errorin=strings.TrimSpace(in)chunks:=strings.SplitN(in," ",2)iflen(chunks)!=2returnfmt.Errorf("Syntax sucks: '%s'",in)a.Package=chunks[0]a.Fingerprint=chunks[1][1:len(chunks[1])-1]returnniltypeDMUAstructFingerprintstringUidstringAllowedPackages[]AllowedPackage control:"Allow" delim:"," funcmain()resp,err:=http.Get("http://metadata.ftp-master.debian.org/dm.txt")iferr!=nilpanic(err)decoder,err:=control.NewDecoder(resp.Body,nil)iferr!=nilpanic(err)fordmua:=DMUAiferr:=decoder.Decode(&dmua);err!=niliferr==io.EOFbreakpanic(err)fmt.Printf("The DM %s is allowed to upload:\n",dmua.Uid)for_,allowedPackage:=rangedmua.AllowedPackagesfmt.Printf(" %s [granted by %s]\n",allowedPackage.Package,allowedPackage.Fingerprint)
Output (truncated!) looks a bit like:
...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM <aelmahmoudy@sabily.org> is allowed to upload:
covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
...
...
deb
Next up, we've got the deb module. This contains code to handle reading
Debian 2.0 .deb files. It contains a wrapper that will parse the control
member, and provide the data member through the
archive/tar interface.
Here's an example of how to read a .deb file, access some metadata, and
iterate over the tar archive, and print the filenames of each of the
entries.
dependency
The dependency package provides an interface to parse and compute
dependencies. This package is a bit odd in that, well, there's no other
library that does this. The issue is that there are actually two different
parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev)
and another in C (in dpkg).
To date, this has resulted in me filing
threedifferentbugs.
I also found a broken package in the
archive,
which actually resulted in another bug being (totally accidentally)
already fixed.
I hope to continue to run the archive through my parser in hopes of finding
more bugs! This package is a bit complex, but it basically just returns what
amounts to be an AST
for our Dependency lines. I'm positive there are bugs, so file them!
funcmain()dep,err:=dependency.Parse("foo bar, baz, foobar [amd64] bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")iferr!=nilpanic(err)anySparc,err:=dependency.ParseArch("sparc")iferr!=nilpanic(err)for_,possi:=rangedep.GetPossibilities(*anySparc)fmt.Printf("%s (%s)\n",possi.Name,possi.Arch)
Gives the output:
foo (<nil>)
baz (<nil>)
fnord (armhf)
version
Right off the bat, I'd like to thank
Michael Stapelberg for letting me graft this
out of dcs and into the go-debian package.
This was nearly entirely his work (with a one or two line function I added
later), and was amazingly helpful to have. Thank you!
This module implements Debian version comparisons and parsing, allowing for
sorting in lists, checking to see if it's native or not, and letting the
programmer to implement smart(er!) logic based on upstream (or Debian)
version numbers.
This module is extremely easy to use and very straightforward, and not worth
writing an example for.
Final thoughts
This is more of a "Yeah, OK, this has been useful enough to me at this point
that I'm going to support this" rather than a "It's stable!" or even
"It's alive!" post. Hopefully folks can report bugs and help iterate on
this module until we have some really clean building blocks to build
solid higher level systems on top of. Being able to have multiple libraries
interoperate by relying on go-debian will be a massive ease.
I'm in need of more documentation, and to finalize some parts of the older
sub package APIs, but I'm hoping to be at a "1.0" real soon now.
As nearly anyone who's worked with me will attest to, I've long since
touted nedbat's talk
Pragmatic Unicode, or, How do I stop the pain?
as one of the most foundational talks, and required watching for all programmers.
The reason is because netbat hits on something bigger - something more
fundamental than how to handle Unicode -- it's how to handle data which is
relative.
For those who want the TL;DR, the argument is as follows:
Facts of Life:
Computers work with Bytes. Bytes go in, Bytes go out.
Relative Data
I've started to think more about why we do the things we do when we write
code, and one thing that continues to be a source of morbid schadenfreude
is watching code break by failing to handle Unicode right. It's hard! However,
watching what breaks lets you gain a bit of insight into how the author
thinks, and what assumptions they make.
When you send someone Unicode, there are a lot of assumptions that have to be
made. Your computer has to trust what you (yes, you!) entered into your web
browser, your web browser has to pass that on over the network (most of the
time without encoding information), to a server which reads that bytestream,
and makes a wild guess at what it should be. That server might save it to a
database, and interpolate it into an HTML template in a different encoding
(called Mojibake), resulting
in a bad time for everyone involved.
Everything's awful, and the fact our computers can continue to display
text to us is a goddamn miracle. Never forget that.
When it comes down to it, when I see a byte sitting on a page, I don't know
(and can't know!) if it's Windows-1252, UTF-8, Latin-1, or EBCDIC. What's
a poem to me is terminal garbage to you.
Over the years, hacks have evolved. We have
magic numbers,
and plain ole' hacks to just guess based on the content. Of course, like
all good computer programs, this has lead to its fair share of hilarious
bugs,
and there's nothing stopping files from (validly!) being multiple things at the
same time.
Like many things, it's all in the eye of the beholder.
Timezones
Just like Unicode, this is a word that can put your friendly neighborhood
programmer into a series of profanity laden tirades. Go find one in the wild,
and ask them about what they think about timezone handling bugs they've seen.
I'll wait. Go ahead.
Rants are funny things. They're fun to watch. Hilarious to give. Sometimes
just getting it all out can help. They can tell you a lot about the true
nature of problems.
It's funny to consider the isomorphic nature of Unicode rants and Timezone
rants.
I don't think this is an accident.
U n i c o d e timezone Sandwich
Ned's Unicode Sandwich applies -- As early as we can, in the lowest level
we can (reading from the database, filesystem, wherever!), all datetimes
must be timezone qualified with their correct timezone. Always. If you mean
UTC, say it's in UTC.
Treat any unqualified datetimes as "bytes". They're not to be trusted.
Never, never, never trust 'em. Don't
process any datetimes until you're sure they're in the right timezone.
This lets the delicious inside of your datetime sandwich handle timezones
with grace, and finally, as late as you can, turn it back into bytes
(if at all!). Treat locations as tzdb entries, and qualify datetime
objects into their absolute timezone (EST, EDT, PST, PDT)
It's not until you want to show the datetime to the user again should you
consider how to re-encode your datetime to bytes. You should think about
what flavor of bytes, what encoding -- what timezone -- should I be
encoding into?
TEST
Just like Unicode, testing that your code works with datetimes is important.
Every time I think about how to go about doing this, I think about that
one time that mjg59 couldn't book a flight
starting Tuesday from AKL, landing in HNL on Monday night, because
United couldn't book the last leg to SFO. Do you ever assume dates only go
forward as time goes on? Remember timezones.
Construct test data, make sure someone in New Zealand's
+13:45 can correctly talk with
their friends in
Baker Island's -12:00,
and that the events sort right.
Just because it's Noon on New Years Eve in England doesn't mean it's not
1 AM the next year in New Zealand. Places a few miles apart may go on Daylight
savings different days. Indian Standard Time is not even aligned on the hour
to GMT (+05:30)!
Test early, and test often. Memorize a few timezones, and challenge
your assumptions when writing code that has to do with time. Don't use
wall clocks to mean monotonic time. Remember there's a whole world out there,
and we only deal with part of it.
It's also worth remembering, as Andrew Pendleton
pointed out to me, that it's possible that a datetime isn't even unique for a
place, since you can never know if 2016-11-06 01:00:00 in America/New_York
(in the tzdb) is the first one, or second one. Storing EST or EDT along
with your datetime may help, though!
Pitfalls
Improper handling of timezones can lead to some interesting things, and failing
to be explicit (or at least, very rigid) in what you expect will lead to an
unholy class of bugs we've all come to hate. At best, you have confused
users doing math, at worst, someone misses a critical event, or our
security code fails.
I recently found what I regard to be a pretty bad
bug in apt (which David has prepared a
fix
for and is pending upload, yay! Thank you!), which boiled down to documentation
and code expecting datetimes in a timezone, but accepting any timezone, and
silently treating it as UTC.
The solution is to hard-fail, which is an interesting choice to me (as a vocal
fan of timezone aware code), but at the least it won't fail by
misunderstanding what the server is trying to communicate, and I do understand
and empathize with the situation the apt maintainers are in.
Final Thoughts
Overall, my main point is although most modern developers know how to deal
with Unicode pain, I think there is a more general lesson to learn -- namely,
you should always know what data you have, and always remember what it is.
Understand assumptions as early as you can, and always store them with the data.
Back in 2014, Mako ran a Boston Iron Blogger chapter, where you had to blog once a week, or you owed $5 into the pot. A while later, I ran it (along with Molly and Johns), and things were great.
When I moved to DC, I had already talked with Tom Lee and Eric Mill about running a DC Iron Blogger chapter, but it hasn t happened in the year and a half I ve been in DC.
This week, I make good on that, with a fantastic group set up at dc.iron-blogger.com; with more to come (I m sure!).
Looking forward to many parties and though provoking blog posts in my future. I m also quite pleased I ll be resuming my blogging. Hi, again, planet Debian!
(You can Follow or Tweet about this blog on Twitter)
Over the last week, people have been saying a lot about the wonderful life of Ian Murdock and his contributions to Debian and the world of free software. According to one news site, a San Francisco police officer, Grace Gatpandan, has been doing the opposite, starting a PR spin operation, leaking snippets of information about what may have happened during Ian's final 24 hours. Sadly, these things are now starting to be regurgitated without proper scrutiny by the mainstream press (note the erroneous reference to SFGate with link to SFBay.ca, this is British tabloid media at its best).
The report talks about somebody (no suggestion that it was even Ian) "trying to break into a residence". Let's translate that from the spin-doctor-speak back to English: it is the silly season, when many people have a couple of extra drinks and do silly things like losing their keys. "a residence", or just their own home perhaps? Maybe some AirBNB guest arriving late to the irritation of annoyed neighbours? Doesn't the choice of words make the motive sound so much more sinister? Nobody knows the full story and nobody knows if this was Ian, so snippets of information like this are inappropriate, especially when somebody is deceased.
Did they really mean to leave people with the impression that one of the greatest visionaries of the Linux world was also a cat burglar? That somebody who spent his life giving selflessly and generously for the benefit of the whole world (his legacy is far greater than Steve Jobs, as Debian comes with no strings attached) spends the Christmas weekend taking things from other people's houses in the dark of the night? The report doesn't mention any evidence of a break-in or any charges for breaking-in.
If having a few drinks and losing your keys in December is such a sorry state to be in, many of us could potentially be framed in the same terms at some point in our lives. That is one of the reasons I feel so compelled to write this: somebody else could be going through exactly the same experience at the moment you are reading this. Any of us could end up facing an assault as unpleasant as the tweets imply at some point in the future. At least I can console myself that as a privileged white male, the risk to myself is much lower than for those with mental illness, the homeless, transgender, Muslim or black people but as the tweets suggest, it could be any of us.
The story reports that officers didn't actually come across Ian breaking in to anything, they encountered him at a nearby street corner. If he had weapons or drugs or he was known to police that would have almost certainly been emphasized. Is it right to rush in and deprive somebody of their liberties without first giving them an opportunity to identify themselves and possibly confirm if they had a reason to be there?
The report goes on, "he was belligerent", "he became violent", "banging his head" all by himself. How often do you see intelligent and successful people like Ian Murdock spontaneously harming themselves in that way? Can you find anything like that in any of the 4,390 Ian Murdock videos on YouTube? How much more frequently do you see reports that somebody "banged their head", all by themselves of course, during some encounter with law enforcement? Do police never make mistakes like other human beings?
If any person was genuinely trying to spontaneously inflict a head injury on himself, as the police have suggested, why wouldn't the police leave them in the hospital or other suitable care? Do they really think that when people are displaying signs of self-harm, rounding them up and taking them to jail will be in their best interests?
Now, I'm not suggesting this started out with some sort of conspiracy. Police may have been at the end of a long shift (and it is a disgrace that many US police are not paid for their overtime) or just had a rough experience with somebody far more sinister. On the other hand, there may have been a mistake, gaps in police training or an inappropriate use of a procedure that is not always justified, like a strip search, that causes profound suffering for many victims.
A select number of US police forces have been shamed around the world for a series of incidents of extreme violence in recent times, including the death of Michael Brown in Ferguson, shooting Walter Scott in the back, death of Freddie Gray in Baltimore and the attempts of Chicago's police to run an on-shore version of Guantanamo Bay. Beyond those highly violent incidents, the world has also seen the abuse of Ahmed Mohamed, the Muslim schoolboy arrested for his interest in electronics and in 2013, the suicide of Aaron Swartz which appears to be a direct consequence of the "Justice" department's obsession with him.
What have the police learned from all this bad publicity? Are they changing their methods, or just hiring more spin doctors? If that is their response, then doesn't it leave them with a cruel advantage over those people who were deceased?
Isn't it standard practice for some police to simply round up anybody who is a bit lost and write up a charge sheet for resisting arrest or assaulting an officer as insurance against questions about their own excessive use of force?
When British police executed Jean Charles de Menezes on a crowded tube train and realized they had just done something incredibly outrageous, their PR office went to great lengths to try and protect their image, even photoshopping images of Menezes to make him look more like some other suspect in a wanted poster. To this day, they continue to refer to Menezes as a victim of the terrorists, could they be any more arrogant? While nobody believes the police woke up that morning thinking "let's kill some random guy on the tube", it is clear they made a mistake and like many people (not just police), they immediately prioritized protecting their reputation over protecting the truth.
Nobody else knows exactly what Ian was doing and exactly what the police did to him. We may never know. However, any disparaging or irrelevant comments from the police should be viewed with some caution.
The horrors of incarceration
It would be hard for any of us to understand everything that an innocent person goes through when detained by the police. The recently released movie about The Stanford Prison Experiment may be an interesting place to start, a German version produced in 2001, Das Experiment, is also very highly respected.
The United States has the largest prison population in the world and the second-highest per-capita incarceration rate. Many, including some on death row, are actually innocent, in the wrong place at the wrong time, without the funds to hire an attorney. The system, and the police and prison officers who operate it, treat these people as packages on a conveyor belt, without even the most basic human dignity. Whether their encounter lasts for just a few hours or decades, is it any surprise that something dies inside them when they discover this cruel side of American society?
Worldwide, there is an increasing trend to make incarceration as degrading as possible. People may be innocent until proven guilty, but this hasn't stopped police in the UK from locking up and strip-searching over 4,500 children in a five year period, would these children go away feeling any different than if they had an encounter with Jimmy Saville or Rolf Harris? One can only wonder what they do to adults.
What all this boils down to is that people shouldn't really be incarcerated unless it is clear the danger they pose to society is greater than the danger they may face in a prison.
What can people do for Ian and for justice?
Now that these unfortunate smears have appeared, it would be great to try and fill the Internet with stories of the great things Ian has done for the world. Write whatever you feel about Ian's work and your own experience of Debian.
While the circumstances of the final tweets from his Twitter account are confusing, the tweets appear to be consistent with many other complaints about US law enforcement. Are there positive things that people can do in their community to help reduce the harm?
Sending books to prisoners (the UK tried to ban this) can make a difference. Treat them like humans, even if the system doesn't.
Recording incidents of police activities can also make a huge difference, such as the video of the shooting of Walter Scott or the UK police making a brutal unprovoked attack on a newspaper vendor. Don't just walk past a situation and assume everything is under control. People making recordings may find themselves in danger, it is recommended to use software that automatically duplicates each recording, preferably to the cloud, so that if the police ask you to delete such evidence, you can let them watch you delete it and still have a copy.
Can anybody think of awards that Ian Murdock should be nominated for, either in free software, computing or engineering in general? Some, like the prestigious Queen Elizabeth Prize for Engineering can't be awarded posthumously but others may be within reach. Come and share your ideas on the debian-project mailing list, there are already some here.
Best of all, Ian didn't just build software, he built an organization, Debian. Debian's principles have helped to unite many people from otherwise different backgrounds and carry on those principles even when Ian is no longer among us. Find out more, install it on your computer or even look for ways to participate in the project.
Assembly
Combine Soylent and Simple Syrup. Create what I m going to start to call Soylent Syrup . Enjoy that one, folks.
Add ice to a rocks glass, pour Soylent Syrup over ice. Add Sherry, Vermouth and Campari. Stir. Garnish with an orange twist.
Big thanks to Matthew Garrett for sparking this one.
A friend recently commented on my rather unusual portrait on my (out of date) page on the Berkman website. Here s the story.
I joined Berkman as a fellow with a fantastic class of fellows that included, among many other incredibly accomplished people, Vivek Kundra: first Chief Information Officer of the United States. At Berkman, all the fellows are all asked for photos and Vivek apparently sent in his official government portrait.
You are probably familiar with the genre. In the US at least, official government portraits are mostly pictures of men in dark suits, light shirts, and red or blue ties with flags draped blurrily in the background.
Not unaware of the fact that Vivek sat right below me on the alphabetically sorted Berkman fellows page, a small group that included Paul Tagliamonte very familiar with the genre from his work with government photos in Open States decided to create a government portrait of me using the only flag we had on hand late one night.
The result shown in the screenshot above and in the WayBack Machine was almost entirely unnoticed (at least to my knowledge) but was hopefully appreciated by those who did see it.
Hello, World
Been a while since my last blog post - things have been a bit hectic lately, and I ve not really had the time.
Now that things have settled down a bit I m in DC! I ve moved down south to join the rest of my colleagues at Sunlight to head up our State & Local team.
Leaving behind the brilliant Free Software community in Boston won t be easy, but I m hoping to find a similar community here in DC.
For the tl;dr: Docker FDW is a thing.
Star it, hack it, try it out. File bugs, be happy. If you want to see what it's
like to read, there's some example SQL down below.
The question is first, what the heck is a PostgreSQL Foreign Data Wrapper?
PostgreSQL Foreign Data Wrappers are plugins that allow C libraries
to provide an adaptor for PostgreSQL to talk to an external database.
Some folks have used this to wrap stuff like
MongoDB, which I always found
to be hilarous (and an epic hack).
Enter Multicorn
During my time at PyGotham, I saw a talk from
Wes Chow about something called
Multicorn. He was showing off some really neat
plugins, such as the git revision history of CPython, and parsed logfiles
from some stuff over at Chartbeat. This basically blew my mind.
All throughout the talk I was coming up with all sorts of things that I wanted
to do -- this whole library is basically exactly what I've been dreaming
about for years. I've always wanted to provide a SQL-like interface
into querying API data, joining data cross-API using common crosswalks,
such as using Capitol Words to query for
Legislators, and use the
bioguide ids
to JOIN against the congress api
to get their Twitter account names.
My first shot was to Multicorn the new
Open Civic Data API I was working on, chuckled
and put it aside as a really awesome hack.
Enter Docker
It wasn't until tianon connected the dots for me
and suggested a Docker FDW did I get really excited.
Cue a few hours of hacking, and I'm proud to say -- here's
Docker FDW.
This lets us ask all sorts of really interesting questions out of the API,
and might even help folks writing webapps avoid adding too much Docker-aware
logic. Abstractions can be fun!
Setting it up
I'm going to assume you have a working Multicorn, PostgreSQL and Docker setup
(including adding the postgres user to the docker group)
So, now let's pop open a psql session. Create a database (I called mine
dockerfdw, but it can be anything), and let's create some tables.
Before we create the tables, we need to let PostgreSQL know where our
objects are. This takes a name for the server, and the Python importable
path to our FDW.
Now that we have the server in place, we can tell PostgreSQL to create a table
backed by the FDW by creating a foreign table. I won't go too much into the
syntax here, but you might also note that we pass in some options - these are
passed to the constructor of the FDW, letting us set stuff like the Docker
host.
And, now that we have tables in place, we can try to learn something about the
Docker containers. Let's start with something fun - a join from containers
to images, showing all image tag names, the container names and the ip of the
container (if it has one!).
OK, let's see if we can bring this to the next level now. I finally got around
to implementing INSERT and DELETE operations, which turned out to be
pretty simple to do. Check this out:
DELETEFROMdocker_containers;
DELETE1
This will do a stop + kill after a 10 second hang behind the scenes. It's
actually a lot of fun to spawn up a container and terminate it from
PostgreSQL.
id
------------------------------------------------------------------
0a903dcf5ae10ee1923064e25ab0f46e0debd513f54860beb44b2a187643ff05
INSERT 0 1
(1 row)
Spawning containers works too - this is still very immature and not super
practical, but I figure while I'm showing off, I might as well go all the way.
Success! This is just a taste of what's to come, so please feel free to hack on
Docker FDW,
tweet me @paultag, file bugs / feature requests.
It's currently a bit of a hack, and it's something that I think has
long-term potential after some work goes into making sure that this is a rock
solid interface to the Docker API.
I ll be there this year!
Talks look amazing, I can t wait to hit up all the talks. Looks really well organized! Talk schedule has a bunch that I want to hit, I hope they re recorded to watch later!
If anyone s heading to PyGotham, let me know, I ll be there both days, likely floating around the talks.
I ll be giving a short talk on Debian and Docker!
I ll prepare some slides to give a brief talk about Debian and Docker, then open it up to have a normal session to talk over what Docker is and isn t, and how we can use it in Debian better.
Hope to see y all in Portland!
Why oh why are they so hard to write?
Even using the built in modules it is insanely hard to debug. Playing a bootsplash in X sucks and my machine boots too fast to test it on reboot.
Basically, euch. All I wanted was a hackers zebra on boot :(
PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types
of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in
PKCS#11 mode using OpenSC to handle system login. 
Ingredients
Ingredients
A friend recently commented on my rather unusual portrait on 
